Let me begin by stating that my judgement is based on assuming the the derivations are mathematically correct - though I attempted at verifying, I am not certain if everything is free of error. With that in mind: Originality: I believe the central idea of the work is novel, and the derivation is unseen, at least to my limited knowledge of related work. However, the generative model involved (AIR) is borrowed directly from related work, and there have definitely been prior work attempting to regularize the posterior. Though the idea of using structural knowledge to directly shape the distribution of interpretable posteriors such as size and number of objects is novel, I am not so sure if this idea is generalizable enough (see quality and significance section) to warrant a major original contribution. Quality: I believe the derivation is technically sound, though I have not verified the mathematical details and cannot be sure if it is free of errors.

I thank the authors for their submission. The paper presents a framework for amortized regularization of the posteriors of deep generative models. I strongly encourage the authors to take into account the reviewers' comments and concerns for the final manuscript.

Deep generative models (DGMs) have shown promise in image generation. However, most of the existing methods learn a model by simply optimizing a divergence between the marginal distributions of the model and the data, and often fail to capture rich structures, such as attributes of objects and their relationships, in an image. Human knowledge is a crucial element to the success of DGMs to infer these structures, especially in unsupervised learning. In this paper, we propose amortized structural regularization (ASR), which adopts posterior regularization (PR) to embed human knowledge into DGMs via a set of structural constraints. We derive a lower bound of the regularized log-likelihood in PR and adopt the amortized inference technique to jointly optimize the generative model and an auxiliary recognition model for inference efficiently. Empirical results show that ASR outperforms the DGM baselines in terms of inference performance and sample quality.

Deep generative models (DGMs) have shown promise in image generation. However, most of the existing methods learn a model by simply optimizing a divergence between the marginal distributions of the model and the data, and often fail to capture rich structures, such as attributes of objects and their relationships, in an image. Human knowledge is a crucial element to the success of DGMs to infer these structures, especially in unsupervised learning. In this paper, we propose amortized structural regularization (ASR), which adopts posterior regularization (PR) to embed human knowledge into DGMs via a set of structural constraints. We derive a lower bound of the regularized log-likelihood in PR and adopt the amortized inference technique to jointly optimize the generative model and an auxiliary recognition model for inference efficiently.